Converting to Optimization in Machine Learning: Perturb-and-MAP, Differential Privacy, and Program Synthesis

On a mathematical level, most computational problems encountered in machine learning are instances of one of four abstract, fundamental problems: sampling, integration, optimization, and search. Thanks to the rich history of the respective mathematical fields, disparate methods with different properties have been developed for these four problem classes. As a result it can be beneficial to convert a problem from one abstract class into a problem of a different class, because the latter might come with insights, techniques, and algorithms well suited to the particular problem at hand. In particular, this thesis contributes four new methods and generalizations of existing methods for converting specific non-optimization machine learning tasks into optimization problems with more appealing properties. The first example is partition function estimation (an integration problem), where an existing algorithm -- the Gumbel trick -- for converting to the MAP optimization problem is generalized into a more general family of algorithms, such that other instances of this family have better statistical properties. Second, this family of algorithms is further generalized to another integration problem, the problem of estimating Rényi entropies. The third example shows how an intractable sampling problem arising when wishing to publicly release a database containing sensitive data in a safe ("differentially private") manner can be converted into an optimization problem using the theory of Reproducing Kernel Hilbert Spaces. Finally, the fourth case study casts the challenging discrete search problem of program synthesis from input-output examples as a supervised learning task that can be efficiently tackled using gradient-based optimization. In all four instances, the conversions result in novel algorithms with desirable properties. In the first instance, new generalizations of the Gumbel trick can be used to construct statistical estimators of the partition function that achieve the same estimation error while using up to 40% fewer samples. The second instance shows that unbiased estimators of the Rényi entropy can be constructed in the Perturb-and-MAP framework. The main contribution of the third instance is theoretical: the conversion shows that it is possible to construct an algorithm for releasing synthetic databases that approximate databases containing sensitive data in a mathematically precise sense, and to prove results about their approximation errors. Finally, the fourth conversion yields an algorithm for synthesising program source code from input-output examples that is able to solve test problems 1-3 orders of magnitude faster than a wide range of baselines

Differentially Private Database Release via Kernel Mean Embeddings

We lay theoretical foundations for new database release mechanisms that allow third-parties to construct consistent estimators of population statistics, while ensuring that the privacy of each individual contributing to the database protected. The proposed framework rests on two main ideas. First, releasing (an estimate of) the kernel mean embedding of the data generating random variable instead of the database itself still allows third-parties to construct consistent estimators of a wide class of population statistics. Second, the algorithm can satisfy the definition of differential privacy by basing the released kernel mean embedding on entirely synthetic data points, while controlling accuracy through the metric available in a Reproducing Kernel Hilbert Space. We describe two instantiations of the proposed framework, suitable under different scenarios, and prove theoretical results guaranteeing differential privacy of the resulting algorithms and the consistency of estimators constructed from their outputs

Determination of fire and explosion characteristics of dust

The aim of this paper is to approximate danger of dust clouds normally occur by determining their explosion characteristics. Nowadays, dusty environment is phenomenon in the industry. In general, about 70% of dust produced is explosive. Dust reduction in companies is the main purpose of the national and European legislative. Early identification and characterization of dust in companies may reduce the risk of explosion. It could be used to identify hazards in industrial production where an explosive dust is produced. For this purpose several standards for identification and characterization of explosion characteristics of industrial dust are being used

Inductive Program Synthesis Over Noisy Data

We present a new framework and associated synthesis algorithms for program synthesis over noisy data, i.e., data that may contain incorrect/corrupted input-output examples. This framework is based on an extension of finite tree automata called {\em weighted finite tree automata}. We show how to apply this framework to formulate and solve a variety of program synthesis problems over noisy data. Results from our implemented system running on problems from the SyGuS 2018 benchmark suite highlight its ability to successfully synthesize programs in the face of noisy data sets, including the ability to synthesize a correct program even when every input-output example in the data set is corrupted

Mixed reality control center for ROVs

U ovom radu dizajniran je i implementiran upravljački sustav za BlueROV2 robotsko vozilo. Implementacijom je realizirana komunikacija robota sa upravljačkim računalom sa kojim komunicira preko ROS-a, šalje tom računalu podatke sa kamere i senzora na robotu, a preko upravljačkog računala primamo input sa joystick-a i upravljamo robotom. Nadalje, implementirana je Unity aplikacija upravljačkog centra u kojoj operator robota u aplikaciji virtualne stvarnosti ima upravljački centar u kojem mu se prikazuje video stream sa kamere robota, te dubina na kojoj je robot i njegova orijentacija u prostoru.In this paper, a control system for the BlueROV2 remotely operated vehicle (ROV) has been designed and implemented. The implementation enables communication between the robot and a control computer via ROS, where data from the robot's camera and sensors are sent to the control computer. Input from the joystick is received through the control computer, allowing control of the robot. Furthermore, a Unity application for the control center has been developed, providing the robot operator with a virtual reality (VR) interface. The VR control center displays a video stream from the robot's camera, as well as the robot's depth and orientation in space

Mixed reality control center for ROVs

U ovom radu dizajniran je i implementiran upravljački sustav za BlueROV2 robotsko vozilo. Implementacijom je realizirana komunikacija robota sa upravljačkim računalom sa kojim komunicira preko ROS-a, šalje tom računalu podatke sa kamere i senzora na robotu, a preko upravljačkog računala primamo input sa joystick-a i upravljamo robotom. Nadalje, implementirana je Unity aplikacija upravljačkog centra u kojoj operator robota u aplikaciji virtualne stvarnosti ima upravljački centar u kojem mu se prikazuje video stream sa kamere robota, te dubina na kojoj je robot i njegova orijentacija u prostoru.In this paper, a control system for the BlueROV2 remotely operated vehicle (ROV) has been designed and implemented. The implementation enables communication between the robot and a control computer via ROS, where data from the robot's camera and sensors are sent to the control computer. Input from the joystick is received through the control computer, allowing control of the robot. Furthermore, a Unity application for the control center has been developed, providing the robot operator with a virtual reality (VR) interface. The VR control center displays a video stream from the robot's camera, as well as the robot's depth and orientation in space

Mixed reality control center for ROVs

U ovom radu dizajniran je i implementiran upravljački sustav za BlueROV2 robotsko vozilo. Implementacijom je realizirana komunikacija robota sa upravljačkim računalom sa kojim komunicira preko ROS-a, šalje tom računalu podatke sa kamere i senzora na robotu, a preko upravljačkog računala primamo input sa joystick-a i upravljamo robotom. Nadalje, implementirana je Unity aplikacija upravljačkog centra u kojoj operator robota u aplikaciji virtualne stvarnosti ima upravljački centar u kojem mu se prikazuje video stream sa kamere robota, te dubina na kojoj je robot i njegova orijentacija u prostoru.In this paper, a control system for the BlueROV2 remotely operated vehicle (ROV) has been designed and implemented. The implementation enables communication between the robot and a control computer via ROS, where data from the robot's camera and sensors are sent to the control computer. Input from the joystick is received through the control computer, allowing control of the robot. Furthermore, a Unity application for the control center has been developed, providing the robot operator with a virtual reality (VR) interface. The VR control center displays a video stream from the robot's camera, as well as the robot's depth and orientation in space